Mobile device avatar generation for biofeedback to customize movement control

ABSTRACT

A system has a sensor that senses baseline sensor data corresponding to one or more baseline movements of a user and senses dynamic sensor data corresponding to one or more dynamic movements of the user. The dynamic sensor data is captured after the baseline sensor data. Furthermore, the system has an image capture device integrated within a mobile computing device. The image capture device captures baseline image capture data corresponding to the one or more baseline movements of the user and captures dynamic image capture data corresponding to the one or more dynamic movements of the user. The dynamic image capture data is captured after the baseline image capture data. Additionally, the mobile computing device is in operable communication with the sensor.

BACKGROUND 1. Field

This disclosure generally relates to computing systems. Moreparticularly, the disclosure relates to the field of biomechanicalvisualization systems.

2. General Background

Sports performance experts (e.g., coaches, trainers, etc.) often attemptto work with athletes to maximize athletic performance for a particularathlete in a specific sport, preempt potential sports-related injuries,and/or rehabilitate the athlete from existing injuries. A commonapproach is for the sports performance expert to provide an in-personphysical assessment of the athlete; yet, such an assessment is ofteninaccurate. Specifically, the sports performance expert is left guessingvarious metrics (e.g., fatigue, force production, angular jointarticulation) generated by the athlete through various exercises, andthen attempts to improve the performance of the athlete based on thoseestimated metrics. In essence, the sports performance expert performs asubjective assessment, which may not take into account invisible metrics(e.g., fatigue) or metrics that are difficult for a human to capture(e.g., joint articulation angles) with any degree of precision and/orconsistency. As a result, the analysis performed by a sports performanceexpert may be quite subjective, and prone to error.

To alleviate such subjectivity, sports performance experts may employvarious types of equipment to measure biomechanical data in aquantitative, rather than qualitative, manner. However, managing dataoutputted by such equipment is often quite difficult because of thelarge amount of biomechanical data that may be generated by suchequipment. For example, that data has to be analyzed to then generatetraining reports based on the analysis. To perform the analysis andgenerate the reports may often take weeks, or possibly even months; sucha time delay is often too significant to help minimize injury riskfactors that may affect a player in the interim period. Furthermore,even after the foregoing time delay, the resulting report may be errorprone because of a misinterpretation of the biomechanical data by thesports performance expert analyzing the biomechanical data during theanalysis phase. Accordingly, the performance enhancing, injuryprevention, and/or injury rehabilitation exercises prescribed in areport, generated from use of conventional equipment, may lackspecificity with respect to the actual biomechanical issue that actuallyneeds to be addressed.

Given the lack of a systematic approach to evaluating biomechanicaldata, both conventional qualitative and quantitative approaches tosports performance enhancement pose significant concerns. Furthermore,many athletes (e.g., non-professionals such as children) do not haveaccess to a sports performance expert and/or sports performanceequipment, leaving many athletes susceptible to significant risk ofinjury.

SUMMARY

In one embodiment, a system has a sensor that senses baseline sensordata corresponding to one or more baseline movements of a user andsenses dynamic sensor data corresponding to one or more dynamicmovements of the user. The dynamic sensor data is captured after thebaseline sensor data. Furthermore, the system has an image capturedevice integrated within a mobile computing device. The image capturedevice captures baseline image capture data corresponding to the one ormore baseline movements of the user and captures dynamic image capturedata corresponding to the one or more dynamic movements of the user. Thedynamic image capture data is captured after the baseline image capturedata. Additionally, the mobile computing device is in operablecommunication with the sensor.

The system also has a processor that is programmed to generateuser-specific baseline data based upon the baseline sensor data and thebaseline image capture data, generate one or more predeterminedbiomechanical rules according to the user-specific baseline data,generate a virtual avatar based upon the one or more predeterminedbiomechanical rules, and generate one or more cues corresponding to thevirtual avatar based upon non-compliance of one or more real-worldmovements of the user with the one or more predetermined rules. The oneor more real-world movements are determined from the dynamic sensor dataand the dynamic image capture data. The virtual avatar has virtualmovements that are synchronized in real-time with the corresponding oneor more real-world movements. Finally, the system has a memory devicethat stores the user-specific baseline data in a baseline data structurefor access by the processor.

As an alternative, a computer program may have a computer readablestorage device with a computer readable program stored thereon thatimplements the functionality of the aforementioned system. As yetanother alternative, a process that utilizes a processor may implementthe functionality of the aforementioned system.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-mentioned features of the present disclosure will become moreapparent with reference to the following description taken inconjunction with the accompanying drawings wherein like referencenumerals denote like elements and in which:

FIG. 1 illustrates an avatar generation system that has a mobilecomputing device communicating with a server, through a network.

FIG. 2A illustrates a system configuration for the server illustrated inFIG. 1A.

FIG. 2B illustrates a system configuration for the mobile computingdevice illustrated in FIG. 1A

FIG. 3A illustrates an example of a multi-user environment that may be abasketball court in which a first user wants to perform variousbasketball movements.

FIG. 3B illustrates a graphical user interface (“GUI”) of the mobilecomputing device visually depicting prompts for the first user toperform the sport-specific exercises, such as a squat.

FIG. 4 illustrates an example of the baseline data structure illustratedin FIG. 1.

FIG. 5A illustrates the second user selecting, at a GUI rendered by themobile computing device, the movement recording indicium to begin theprocess of avatar generation based upon real-time, or substantiallyreal-time, data determined from the kinematic motion of the first user.

FIG. 5B illustrates an example of the mobile computing device renderingan avatar, which provides biofeedback regarding the movements of thefirst user in real-time, or substantially real-time, of performance ofthose movements by the first user.

FIG. 5C illustrates an example of the avatar, illustrated in FIG. 5B,concluding a virtual movement that corresponds to a real-world movementof the first user.

FIG. 5D illustrates the mobile computing device automaticallyredirecting the GUI to one or more instructional videos upon thecorrection performed by the first user not complying with the one ormore predetermined biomechanical rules.

FIG. 5E illustrates the GUI rendering a biomechanical analysis displaywith various menu selections.

FIG. 6 illustrates an example of a single-user configuration in whichthe first user performs the movements, while concurrently viewing themobile computing device.

FIG. 7 illustrates a compositing process that may be utilized togenerate the baseline data for the baseline data structure illustratedin FIG. 4.

FIG. 8 illustrates a process that may be utilized to generate a virtualavatar based upon sensor data and imagery data to customize biofeedbackfor the first user.

DETAILED DESCRIPTION

An avatar generation configuration generates a virtual avatar that mayprovide biofeedback to a user (i.e., mirroring the biomechanicalmovements of the user). Rather than providing a standardized overlay,indicating universal ranges of motion, the avatar generationconfiguration generates a customized avatar that uniquely identifiesmovement control for a particular user. In other words, acceptableranges of motion may vary from user-to-user—biomechanical efficiency isnot one-size-fits-all. In particular, the avatar generationconfiguration generates a baseline data structure that is generated by arules generation engine. A user may be prompted, by a mobile computingdevice (e.g., smartphone, tablet device, smart glasses, smart watch,smart bracelet, smart apparel, etc.) or another user operating themobile computing device, to perform a series of movements to capturebaseline data particular to that user (i.e., the user's movementsignature). During this learning phase, the rule generation engine,optionally with an artificial intelligence (“AI”) system, determinesoptimal ranges of movement for the particular user, for which themovement data was captured by the mobile computing device. For example,the mobile computing device may prompt (e.g., audio, video, hapticvibration, etc.) the user, or operator of the mobile computing devicecapturing video of the user, to perform the various baseline movements.Once the baseline data structure is established, the avatar generationconfiguration has the customized set of rules from which biomechanicalanalysis may be performed for the particular user.

In contrast with analyzing static positions (e.g., sitting at a desk) ormovements that involve repetitive motions in a stationary position(e.g., squat), the avatar generation configuration is capable ofanalyzing more complex dynamic movements that are sport-specific (e.g.,a basketball jump shot). Based upon the baseline data structure, theavatar generation configuration is able to determine deviations from thebiomechanical rules in the baseline data structure, and providerecommendations (e.g., exercises) to correct the dynamic movement forthe user, on a customized basis. Furthermore, because the baseline datastructure is established prior to actual analysis of the user's dynamicmovements, the baseline rules are available for fast access whenperforming biomechanical analysis of a user performing a particularsport-specific movement, thereby allowing for real-time assessment of auser's dynamic movement and real-time correction of that that dynamicmovement.

The real-time correction may be provided via an avatar, corresponding tothe user performing the dynamic movement, rendered on the displaydevice. Notably, the avatar generation configuration generates theavatar based on image data of the user, which may be captured by animage capture device integrated within the mobile computing device or adistinct image capture device that is in operable communication with themobile computing device, in conjunction with sensor data of the user,which may be captured by one or more sensors positioned on the user orin proximity to the user. Furthermore, the avatar generationconfiguration may analyze the image data to determine kinematic motionalong one plane, while analyzing the sensor data to determine kinematicmotion along a distinct plane. For example, for the purpose ofdetermining movement control along the transverse plane (e.g., chestmovement, sacrum movement, thigh movement, etc.), a sensor may besignificantly more accurate than a camera, which is limited in itsability to fully capture movement along the transverse plan when viewingthe user along the frontal plane, which is a typical viewpoint from amobile computing device; conversely, the camera may be more accurate incapturing data along the frontal plane given its direct viewpoint asopposed to computer vision processes utilized in conjunction with one ormore sensors to deduce imagery. Accordingly, the avatar generationconfiguration may obtain imagery data for multiple planes, but filterthe image data for the plane that is optimal for an image capturedevice. Similarly, the avatar generation configuration may obtain sensordata for multiple planes, but filter the sensor data for the plane thatis optimal for a sensor. Therefore, the avatar generation configurationselects data from a particular data source (e.g., image capture device,sensor, etc.) based on a particular plane and the data source that isoptimal for that plane. As a result, the avatar generation configurationis able to provide for improved accuracy in detecting kinematic motionover conventional configurations.

Furthermore, the avatar generation configuration may providebiomechanical feedback to the user via the avatar. For example, if theuser should correct a particular movement of a limb, the GUI of themobile computing device may provide a visual indication on, or inproximity to, the avatar. As a result, the user or an additional useroperating the mobile computing device may determine in real-time(measured as a humanly imperceptible time difference between themovement of the user and the generation of the avatar), or substantiallyreal-time (measured as a noticeable time difference that is perceived asan insignificant time lag (e.g., one to five seconds)), biomechanicalcorrections to the movement of the user during that movement, asrendered by the avatar. If another user views the avatar, that otheruser can then effectively coach the user performing the movement as tohow to correct the biomechanical movement. Conversely, if the userperforming the movement views the mobile computing device (e.g., holdingthe mobile computing device during viewing, wearing the mobile computingdevice during viewing, positioning the mobile computing device on atripod, supporting the mobile computing device with an accessory such asa kickstand, etc.), he or she may be able to self-correct the movementduring performance of the movement, or shortly thereafter.

Additionally, the GUI rendered at the mobile computing device mayprovide other resources to assist the user performing the movement withbiomechanical correction. For example, the GUI may display abiomechanical assessment particular to the user performing the movement.The biomechanical assessment may be used by performance enhancementcoach, a friend, or the user performing the movement to pinpointspecific biomechanical deficiencies for correction. As another example,the GUI may display various instructional videos and/or output variousinstructional audio emissions for specific exercises that arerecommended by the avatar generation configuration, potentially via theAI, to enhance the movement control of the user. For example, the avatargeneration configuration may recommend a squat exercise for improvingparticular kinematic motions associated with a dynamic movement, such asa jump shot. The GUI on the mobile computing device may then provide avideo and/or an audio instructional as to how to properly perform thesquat. (A squat is just one of many examples of possible exercises thatmay be recommended, given that many other types of exercises may berecommended to resolve a particular biomechanical deficiency of theuser's movement control.)

In one embodiment, the avatar is a two-dimensional (“2-D”) orthree-dimensional (“3D”) representation of the user performing themovement that may be displayed by a mobile computing device.Alternatively, instead of a mobile computing device, a stationarycomputing device (e.g., desktop computer, kiosk, etc.) may be utilizedto render the avatar. In an alternative embodiment, the avatar may bedisplayed as a hologram that is emitted by a holographic projector inoperable communication with a mobile computing device or a stationarycomputing device.

The avatar generation configuration may be utilized by a wide variety oftypes of athletes (casual to professional, children to adults, solosports to team sports, etc.). Rather than necessitating an in-personsports performance expert and a sports performance lab with extensiveequipment, the avatar generation configuration may be implemented in aphysical environment that is convenient for the user, such as a livingroom where dynamic movements may be potentially simulated, or a physicalenvironment that provides a realistic feel for a sport-specific event,such as a basketball court. Furthermore, the avatar generationconfiguration may be utilized for a variety of movements (e.g.,exercise, sport-specific movement, sport-specific drill, yoga pose,etc.). Instead of waiting a prolonged period of time for data obtainedfrom sports lab equipment in conventional configurations to measure andanalyze data, the avatar generation configuration presents cues to theuser in real-time, or substantially real-time, so that the user mayimmediately make biomechanical corrections when it matters most tomaximize performance and/or prevent injury: during performance of theathletic movement.

FIG. 1 illustrates an avatar generation system 100 that has a mobilecomputing device 101 communicating with a server 102, through a network104. Firstly, the mobile computing device 101 may be utilized to obtainbaseline data customized to the particular movements of a user; viaintegrated componentry within the mobile computing device 101 or one ormore peripheral devices in operable communication therewith. Thebaseline data may be determined based on sensor data and/or imagery dataobtained by the mobile computing device 101. In one embodiment, themobile computing device 101 sends the sensor data and/or the imagerydata through the network 104 to the server 102, which may then utilize arules generation engine 105 to establish one or baseline rules based onthe baseline data. The baseline data may have overlapping data points,the redundancy of which may be removed by the rules generation engine105. For example, image data and sensor data both may include datapertaining to a user's hips, but the rules generation engine 105 mayfilter out the image data, which is captured from the perspective of afrontal plane, to only include the sensor data captured from theperspective of a transverse plane, thereby leading to improved accuracyin detecting kinematic motion and improved computing efficiency byeliminating redundancies. For instance, the server 102 may be inoperable communication with a database 103 that stores a baseline datastructure 107. In essence, the baseline data structure 107 provides forfast and efficient access to one or more predetermined biomechanicalrules corresponding to baseline data customized to a particular user.The server 102 may send the biomechanical rules, via the network 104, tothe mobile computing device 101, which may then determine, based onreal-time sensor data and/or image data, compliance with the one or morepredetermined biomechanical rules, which were determined according topreviously captured image and/or sensor data. For example, the userperforming the movement may have been prompted by the mobile computingdevice 101, or a user operating the mobile computing device 101, toperform one or more baseline movements, which may be directly orindirectly tied to movement control for a sport-specific movement. Forinstance, the user may be prompted to perform a squat movement fordetermining baseline joint angles particular to a user for subsequentperformance of the dynamic movement of a jump shot; alternatively, theuser may be prompted to perform the actual movement of the jump shot todetermine the baseline data for the jump shot. After generation of therules by the rules generation engine 105, the mobile computing device101 may determine real-time compliance with the rules based on thesensor data and image data captured by the mobile computing device 101.Furthermore, the mobile computing device 101 may receive avatar datafrom an avatar generation engine 106 at the server 102. In essence, theavatar is a visual representation specific to the user that hascustomized dimensions (e.g., limb lengths, widths, etc.) and kinematicranges particular to the user, as indicated by the rules of the baselinedata structure. The avatar may be represented in the form of data, or aprogram such as an applet that is accessible by the mobile computingdevice 101 via the server 102. The mobile computing device 101 mayoperate its own computer executable code, or the applet, to displayvisual non-compliance indications on the avatar, or in proximitythereto. (Audio indications or haptic vibration indications may beemitted as an alternative to visual indications.) In this embodiment,the mobile computing device 101 relies on the server 102 as acloud-based configuration to generate the rules and the avatar.

In alternative embodiment, the mobile computing device 101 has the rulesgeneration engine 105, the avatar generation engine 106, and/or thedatabase 103 stored locally thereon, without full reliance on the server102. In other words, the mobile computing device 101 may communicatewith the server 102 in a manner that allows for partial cloud computing.

In essence, the visual or audio indications may be cues that help guidethe user performing the movement toward compliance with thebiomechanical rules established by the rules generation engine 105. Thebiomechanical rules are based on specific ranges of motion (e.g., jointangles, anatomical structure flexion, etc.) that are customized for theuser performing the movement based on the anatomical structure andranges of motion particular to that user, rather than just a general setof ranges that may not be conducive to the particular user performingthe movement. Furthermore, the biomechanical rules are not onlyestablished to account for the maximum ranges of motion of theparticular user, but also ranges of motion that are deemed to be safe.In other words, just because a user can move a limb through a particularrange of motion does not mean that the user should fully move throughthat range, or apply force throughout that entire range. As a result,the rules generation engine 105 may apply a rule factor (e.g., amount ofrange reduction, range multiplier that reduces the range by a certainmultiplier, etc.) to the maximum ranges of motion of a particular userto help minimize the risk of injury for that user, while also trainingthe user to obtain peak performance through his or her range of motionfor particular limbs and/or joints. Upon detecting a deviation from thepredetermined rules (e.g., a deviation from one or more predeterminedranges of motion as determined by a rule factor applied to the ranges ofmotion corresponding to the anatomical structures of a particular user),which are based on the particular user's baseline data and potentialrule factors, the avatar generation configuration 100 may generate thecues to guide the user back toward compliance with the rules. If theavatar generation system 100 determines that the user is incapable ofcomplying with the rules, which may be based on a predetermined timethreshold for rule compliance being exceeded, the avatar generationsystem 100 may generate an interrupt instruction to pause, ortemporarily prevent user access to, the rendering of the avatar untilthe user accesses one or more safety instructional videos or audiorecordings.

FIGS. 2A and 2B illustrate system configurations for the variouscomponentry of the avatar generation system 100 illustrated in FIG. 1.In particular, FIG. 2A illustrates a system configuration for the server102 illustrated in FIG. 1A. The server 102 may have a processor 201,which may be specialized for image analysis, audio analysis, baselinedata composition from imagery data and sensor data, and/or avatargeneration. Accordingly, the processor 201 may be used to perform theoperations illustrated in FIG. 1 for generating an avatar, and potentialcorresponding cues, based on compliance with one or more predeterminedbiomechanical rules.

The system configuration may also include a memory device 202, which maytemporarily store the baseline data structure 107, illustrated in FIG.1, for improved processing times by the processor 201. As a result, theavatar generation system 100 is able to provide real-time, orsubstantially real-time, biomechanical assessment and/or biomechanicalcue correction via the avatar. Furthermore, the memory device 202 maystore computer readable instructions performed by the processor 201. Asan example of such computer readable instructions, a data storage device205 within the system configuration may store biomechanical rulesgeneration code 206 and avatar generation code 207. The processor 201may execute the biomechanical rules generation code 206 to establish thebiomechanical rules as predetermined rules by compositing sensor andimage data into baseline data that is stored in the baseline datastructure 107, which may then be stored by the memory device 202 forreal-time, or substantially real-time, avatar generation. Additionally,the rules generation code 206 may allow the processor 201 to analyzesubsequently received sensor and image data to determine real-timecompliance with the predetermined rules. Furthermore, the processor 201may execute the avatar generation code 207 to automatically generate theavatar, as well as visual and/or audio cues corresponding to the avatar.

Moreover, the system configuration may have one or more input/output(“I/O”) devices 203 that may receive inputs and provide outputs. Variousdevices (e.g., keyboard, microphone, mouse, pointing device, handcontroller, joystick, display device, holographic projector, etc.) maybe used for the I/O devices 203. The system configuration may also havea transceiver 204 to send and receive data. Alternatively, a separatetransmitter and receiver may be used instead.

Conversely, FIG. 2B illustrates a system configuration for the mobilecomputing device 101 illustrated in FIG. 1A. In particular, a processor251, which may be specialized for image rendering, may be used toperform rendering of the avatar associated with the user performing themovement, and corresponding virtual cues.

The system configuration may also include a memory device 252, which maytemporarily store computer readable instructions performed by theprocessor 251. As an example of such computer readable instructions, adata storage device 257 within the system configuration may store GUIgeneration code 258. The processor 251 may execute the GUI generate code258 to generate and/or render a GUI associated with a softwareapplication executed by the mobile computing device 101. The GUI mayinclude an interface that allows for menu selections for biomechanicalassessment and biomechanical correction particular to the userperforming the movement, as well as real-time, or substantiallyreal-time, visual rendering of an avatar associated with the user duringthe performance of the movement by the user. In another embodiment, theavatar generation code 207 is stored by the data storage device 257 ofthe mobile computing device 101 instead of the server 102.

Moreover, the system configuration of the mobile computing device 101may have sensors 253 integrated therein, or may be in operablecommunication with the sensors 253. Furthermore, the one or more sensors253 may be positioned directly on the user to perform variousmeasurements during the user's performance of the movement. For example,the sensors 253 may be inertial measurement units (e.g., accelerometer,gyroscope, magnetometer, etc.) that are utilized to measure movement ofvarious limbs/joints (e.g., knees, ankles, waist, lower back chest,etc.) of the user performing the movement. In essence, the sensors 253may be utilized to detect subtle movements of the user performing themovement along a particular plane, such as the transverse plane. Theprocessor 201 and/or the processor 251 may utilize various computervision processes to generate imagery and perform image analysis on thedata captured by the sensors 253. Additionally, the system configurationof the mobile computing device 101 may have one or more image capturedevices 254 (e.g., camera) integrated therein, or may be in operablecommunication with the one or more image capture devices 254. Incontrast with the sensors 253 that are most suitable for detectingsubtle movements along the transverse plane, the image capture devices254 may be utilized to detect gross motor movements, such as along thefrontal plane. As an example, the mobile computing device 101 may havean integrated image capture device 254, but may be in operablecommunication with one or more external sensors that are positioneddirectly on the user performing the movement via one or more connectionmechanisms (e.g., straps, bands, belts, etc.). In one embodiment, theprocessor 251 of the mobile computing device 101 sends the sensor andimage data to the server 102 so that the server 102 may perform thecompositing of the sensor and image data. In another embodiment, theprocessor 251 of the mobile computing device 101 performs thecompositing of the sensor and imagery data.

In another embodiment, an integrated sensor may be used in place ofdistinct sensors 253 and image capture devices 254. For example, adepth/heat sensing (e.g., thermal imaging) camera may be utilized tocapture imagery and sense movement of a limb of the user. Furthermore,the sensors 253 may be specifically configured to detect not onlykinematic motion, but also biometric data (e.g., heartrate, temperature,etc.); such biometric data may be indicative of fatigue and/or jointinflammation. Accordingly, the one or more sensors 253 may sense/measurea variety of metrics (e.g., velocity, acceleration, force, heat, etc.),potentially independent of, or in conjunction with, the image capturedevices 254.

If external to the mobile computing device 101, the sensors 253 and/orimage capture devices 254 may communicate with the mobile computingdevice 101 via various types of connections (e.g., wired connection orwireless connection such as WiFi, BLUETOOTH, etc.).

Finally, the system configuration of the mobile computing device 101 mayreceive inputs and provide outputs. Various devices (e.g., keyboard,microphone, mouse, pointing device, hand controller, joystick, displayscreen, holographic projector, etc.) may be used for the I/O devices255. If external to the mobile computing device 101, the I/O devices 255may communicate with the mobile computing device 101 via various typesof connections (e.g., wired connection or wireless connection such asWiFi, BLUETOOTH, etc.).

The system configuration may also have a transceiver 256 to send andreceive data. Alternatively, a separate transmitter and receiver may beused instead.

FIGS. 3A and 3B illustrate a multi-user environment 300 in which theavatar generation system 100 may be practically implemented. As anexample, as illustrated in FIG. 3A, the multi-user environment 300 maybe a basketball court in which a first user 301 wants to perform variousbasketball movements. Furthermore, a second user 302 (e.g., friend,coach, trainer, etc.) may operate the mobile computing device 101,illustrated in FIGS. 1 and 2B. The first user 301 may wear a variety ofsensors, such as a chest sensor 311, a waist sensor 310, and anklesensors 309 a and b. (These sensors are provided only as examples.Additional, or alternative, sensors may be positioned on otheranatomical structures of the first user 301, such as the elbows, wrists,knees, head, etc.) Additionally, the second user 302 may operate animage capture device 254 to capture imagery of the movement of the firstuser 301.

Although not visible to the users 301 and 302, FIG. 3A illustrates afrontal plane 304 and a transverse plane 303 to depict movements alongthese different planes; particularly, to illustrate the data filteringspecific to a particular plane for optimal accuracy in generating anavatar.

Furthermore, FIG. 3A illustrates the GUI 305 that is generated by themobile computing device 102. The GUI 305 improves operation of themobile computing device 102 with respect to biomechanical assessment andcorrection. Firstly, the GUI 305 renders a baseline recording indicium307, which allows the second user 302 to obtain baseline data formovements of the first user 301. The first user 301 may be prompted bythe mobile computing device 102 to perform one or more baselinemovements, such as exercises that determine the baseline ranges ofmotion for the particular user 301, which may differ significantly fromthat of another user. After the second user 302 activates the baselinerecording indicium 307, the mobile computing device 101 may emit, viavisual or audio outputs, one or more prompts to request that the firstuser 301 perform the movements that will help generate the baselinedata. As an example, the first user 301 may want to work on his or herbasketball jump shot. Accordingly, the mobile computing device 101 maydetermine certain sport-specific exercises with ranges of motion thatcorrelate to a basketball jump shot, such as squats and lunges.Therefore, upon receiving prompts from the mobile computing device 101,or prompts received from the second user 302 interacting with the mobilecomputing device 101, the first user 301 may perform the baselinemovements while the second user 302 maneuvers the mobile computingdevice 101 to capture video of the first user 301 performing thebaseline exercises. In one embodiment, the GUI 305 displays a video area312 that displays real-time, or substantially real-time, recording ofthe real-world movements of the first user 301.

FIG. 3B illustrates the GUI 305 of the mobile computing device 101visually depicting prompts for the first user 301 to perform thebaseline exercises, such as a squat. In particular, as the first user301 performs the squat motion, the image capture device 254, integratedwithin the mobile computing device 101, and the sensors 309 a, 309 b,310, and 311, worn by the first user 301, detect kinematic motion alongthe frontal plane 304 and the transverse plane 303. However, the imagecapture device 254 is clearly more suited to optimally capture the grossmotor movements along the frontal plane 304; whereas the sensors 309 a,309 b, 310, and 311 are more suited to capture fine motor movementsalong the transverse plane 303.

FIG. 4 illustrates an example of the baseline data structure 107illustrated in FIG. 1. Upon gathering various image and sensor data fromthe image capture device 254 and the sensors 309 a, 309 b, 310, and 311,respectively, the server 102 and/or the mobile computing device 101,illustrated in FIG. 1, may composite baseline data for the first user301. In particular, the server 102 and/or the mobile computing device101 may select the most optimal data source for a given plane, filterout data from other data sources, and composite the resulting data intoa baseline data set that is customized for movement control of the firstuser 301. The baseline data structure 107 may then store a plurality ofbiomechanical rules based on the baseline data set.

As an example, the baseline data structure 107 may have a plurality offields, such as a biomechanical assessment field 401, a variable field402, a baseline range field 403, and a rules field 404. For instance,the biomechanical assessment field 401 may include various biomechanicalassessment metrics (e.g., general strength, force absorption, relativestrength, total body power, movement, and manual mobility). Furthermore,for each biomechanical assessment metric, the baseline data structure107 may store a variable specific to that biomechanical assessmentmetric (e.g., maximum force for general strength, impact force for forceabsorption, ground contact time for relative strength, jump distance fortotal body power, peak torso extensional flexion for movement, and peakshoulder flexion for manual mobility). (Although only one variable isillustrated per biomechanical assessment for illustrative purposes,multiple variables may be stored for each biomechanical assessmentmetric.)

Furthermore, the baseline data structure 107 may store baseline rangedata, which is indicated by the baseline range field 403, that indicatesone or more predetermined ranges of motion specific to the particularuser 301, as determined from the composite baseline data (i.e. optimalsensor data composited with optimal image data). Additionally, thebaseline data structure 107 may indicate one or more biomechanical rulesthat are generated based on the composite baseline data. In oneembodiment, the processor 201 of the server 102 and/or the processor 251of the mobile computing device 101 may generate the biomechanical rulesby imposing a rule factor on the composite baseline data. For instance,allowing for full knee flexion throughout the entirety of thepredetermined range of motion for a particular user may be consideredunsafe for the user 301; whereas reducing the range of motion by apredetermined amount at each endpoint of the range of motion may helppreserve the safety of the user 301 and the longevity of his or herlimbs and joints. For example, an angular displacement of a knee mayallow for a range of motion of one hundred twenty degrees, but a rulemay be generated to reduce that range of motion at one end point to onlyallow for a ninety degree displacement. In certain instances, oneendpoint may be reduced by a particular rule factor whereas anotherendpoint may not be reduced at all.

The baseline data structure 107 may be stored as a variety of datastructures (e.g., two-dimensional array as illustrated in FIG. 4,one-dimensional array, linked list, double linked list, etc.).Furthermore, the baseline data structure 107 may have one or morepointers to other data structures, which may also store baseline data.

FIGS. 5A-5E illustrate an example of avatar generation in asport-specific environment 300 (e.g., basketball) to provide biofeedbackto the first user 301 to customize control movements (e.g., jump shot)particular to that user. For example, FIG. 5A illustrates the seconduser 302 selecting, at GUI 305 rendered by the mobile computing device101, the movement recording indicium 306 to begin the process of avatargeneration based upon real-time, or substantially real-time, datadetermined from the kinematic motion of the first user 301. At thispoint, the mobile computing device 101 may already have the one or morepredetermined biomechanical rules, which were based on the compositedbaseline data that was composited from the senor and image dataillustrated in FIGS. 3A and 3B, as an example. Accordingly, afterselection of the movement recording indicium 306, the mobile computingdevice 101 may begin receiving subsequent sensor and image data,potentially in the same manner that was used to capture data for thepurpose of generating the one or more predetermined rules. FIG. 5Billustrates an example of the mobile computing device 101 rendering anavatar 501, which provides biofeedback regarding the movements of thefirst user 301 in real-time, or substantially real-time, of performanceof those movements by the first user 301 (i.e., the avatar 501 mirrorsthe movements of the first user 301 in a synchronized manner). Theavatar may be displayed in a virtual video display area 520. As anexample, FIG. 5B illustrates the first user 301 beginning to perform ajump shot; concurrently with such movement, the mobile computing device101 renders a display of the avatar 501 also performing that movement.As a result, the second user 302 is able to view an up-close display ofthe particular limb/joint motion of the first user 301, therebyproviding the second user 302 more detailed biomechanical data thanviewing the first user 301 without the mobile computing device 101.Furthermore, FIG. 5C illustrates an example of the avatar 501,illustrated in FIG. 5B, concluding a virtual movement that correspondsto a real-world movement of the first user 301. Moreover, the mobilecomputing device 101 and/or the server 102 may monitor the biomechanicalmovements of the first user 301 to determine compliance with thebiomechanical rules stored by the baseline data structure 107,illustrated in FIG. 4. The mobile computing device 101 may then displaya visual indication 510 (e.g., arrow, marker, flashing light, etc.)and/or emit an audio indication (e.g., voice-based instruction) thatindicates how the first user 301 may perform correction of any movementsthat do not comply the one or more predetermined rules, such as thoseindicated by the rules field 404 of the baseline data structure 107illustrated in FIG. 4. For example, the first user 301 may have landedin such a manner that one of his or her knees moved inward too much in amanner that exceeds an acceptable predetermined range of motion, asdictated by the one or more predetermined biomechanical rules. As aresult, the mobile computing device 101 may generate a correction visualcue 510 that allows the first user 301 to perform correction inreal-time, or substantially real-time, with his or her movement. Inessence, the correction visual cue 510 allows for customized biofeedbackto customize movement control for the particular first user 301.

In one embodiment, as illustrated in FIG. 5D, the mobile computingdevice 101 may automatically redirect the GUI 305 to one or moreinstructional videos 521 and 522 upon the correction performed by thefirst user 301 not complying with the one or more predeterminedbiomechanical rules. For example, the mobile computing device 101 and/orthe server 102 may establish a predetermined time limit in which thefirst user 301 has to comply with the one or more predetermined rules.Should the first user 301 not complete the correction within thatpredetermined time limit, the mobile computing device 101 and/or theserver 102 may deduce that the first user 301 is having some difficulty(e.g., biomechanical, knowledge, etc.) performing the correction.Accordingly, the mobile computing device 101 and/or the server 102 mayautomatically redirect the GUI 305 to an exercise regimen screen toallow the user to watch videos and/or listen to audio instructing thefirst user 301 on exercises to help improve the biomechanical ability ofthe first user 301 to perform the biomechanical correction. In analternative, the second user 302 may manually navigate to the exerciseregimen display and select from the instructional videos and/or audio.For instance, the second user 302 may select the exercise regimenindicium 309 from the GUI 305 illustrated in FIG. 3A.

Moreover, the GUI 305 may display a biomechanical assessment upon aselection of the biomechanical assessment indicium 308. As an example,FIG. 5E illustrates the GUI 305 rendering a biomechanical assessmentdisplay with various menu selections. As examples, individual dataindicium 550, analysis indicium 551, and recommendations indicium 552may be displayed by the GUI 305. Furthermore, each indicium may havevarious sub-indicia. For example, the analysis indicium 551 may have ageneral strength sub-indicium 553, a force absorption sub-indicium 554,a reactive strength sub-indicium 555, a total body power sub-indicium556, a movement sub-indicium 557, and a manual mobility sub-indicium558. Accordingly, the second user 302 may select the various indicia andsub-indicia to obtain particular data pertaining to the movementperformance of the first user 301. In one embodiment, such data includesa plurality of scores (e.g., numeric, relative (low, medium, high),etc.) corresponding to the various metrics. Furthermore, therecommendations indicium 552 may include various recommendations forexercises or modifications to the movements performed by the first user301.

FIGS. 3A, 3B, and 5A-5E are directed to the instance whereby the firstuser 301 performs the baseline exercises/movements, followed by theactual sport-specific movements or other types of movements. In essence,the mobile computing device 101 is illustrated as allowing for amulti-user configuration: the first user 301 performs movements whilethe second user 302 guides the first user 301 based uponprompts/indications rendered by the mobile computing device 101.

In an alternative embodiment, a single-user configuration, rather than amulti-user configuration, may be utilized to implement theconfigurations provided for herein. As an example, FIG. 6 illustrates asingle-user configuration in which the first user 301 performs themovements, while concurrently viewing the mobile computing device 101.As an example, the mobile computing device 101 may be positioned on atripod 601 to allow the first user 301 to view the avatar 501 duringperformance of the movements. (Other types of devices (e.g., kickstands,straps, etc.) may be utilized instead. Alternatively, the first user 301may hold or wear the mobile computing device 101 during the movements.)

Although the avatar 501 is illustrated as being rendered directly on thedisplay screen of the mobile computing device 101, in an alternativeembodiment, the avatar 501 may be rendered external to the mobilecomputing device 101. For example, an integrated holographic projector,or an external holographic projector in operable communication with themobile computing device 101, may emit a hologram of the first user 301as he or she performs the movements. As another example, an externaldisplay device (e.g., television, computer monitor, etc.) may be inoperable communication with the mobile computing device 101, and renderthe movements of the first user 301.

In one embodiment, the avatar 501 may be generated from a skeletal bodymap that is determined from the real-world movements of the first user301 based upon a computer vision process that is performed by the imagecapture device 254 in conjunction with the processor 201 and/or theprocessor 251. The accuracy of the measurements may be improved via thesensors 254. In essence, the skeletal body map may be transferred to theavatar 501 to allow for synchronized movement between the avatar 501 andthe first user 301. In an alternative embodiment, the avatar 501 may begenerated by obtaining a still, or dynamic, real-world image of thefirst user and generating corresponding anatomical structures for theavatar 501 for subsequent manipulation. The avatar 501 provides forclear articulation of the limbs/joints of the first user 301, which maynot necessarily be emphasized from a visual cue perspective when usingreal-world video of the first user 301. Furthermore, the avatar 501 maybe selected from a variety of computer-animated figures (e.g., cartoons,emoji, etc., such as favorite sports players of the first user 301). Asanother alternative, instead of the avatar 501, video of the first user301 may be displayed. As yet another alternative, the avatar 501 may beutilized for the baseline movements, illustrated in FIGS. 3A and 3B, asopposed to the video/imagery of the first user 301.

FIG. 7 illustrates a compositing process 700 that may be utilized togenerate the baseline data for the baseline data structure 107illustrated in FIG. 4. In particular, the mobile computing device 101and/or the server 102 may take the sensor data 701 and the image capturedata 702 as captured from the first user 301 and perform filtering onthat data to optimize data selection based on specific planes. Forinstance, in one embodiment, a plane-specific sensor filter 703 mayfilter the sensor data 701 to select only the sensor data that is mostsuitable for a particular plane (e.g., transverse plane 303) and theimage capture data 702 that is most suitable for another plane (e.g.,frontal plane 304). Accordingly, the compositing process 700 efficientlyoptimizes data selection based upon an optimal data source, and removesdata redundancies by filtering out duplicative data that is less thanoptimal from that data source. For instance, at the completion of thefiltering process, only sensor data pertaining to the hips is utilizedfrom the transverse plane 303; the sensor data pertaining to gross motormotions may be filtered out since the image capture device 254 maycapture that data in a more accurate manner. Conversely, only image datapertaining to the gross movements (e.g., knee flexion) is utilized fromthe frontal plane 304; the image data pertaining to the hips is filteredout since the sensor 253 may capture that data in a more accuratemanner. In another embodiment, the overlapping data may be stored ratherthan filtered, but the optimal data may be utilized for the compositingprocess. After filtering, the plane-specific sensor filter 703 and theplane-specific image capture device filter 704 may send the remainingdata to the compositor 705 that may composite the remaining data forstorage as baseline data in the baseline data field 403 in the baselinedata structure 107.

FIG. 8 illustrates a process 800 that may be utilized to generate avirtual avatar based upon sensor data and imagery data to customizebiofeedback to the first user 301. At a process block 801, the process800 senses, with a sensor 253, baseline sensor data corresponding to oneor more baseline movements of the first user 301. Furthermore, at aprocess block 802, the process 800 senses, with the sensor 253, dynamicsensor data corresponding to one or more dynamic movements of the firstuser 301. In addition, at a process block 803, the process 800 captures,with an image capture device 254 integrated within a mobile computingdevice 101, baseline image capture data corresponding to the one or morebaseline movements of the first user 301. At a process block 804, theprocess 800 captures, with the image capture device 254, dynamic imagecapture data corresponding to the one or more dynamic movements of thefirst user 301. Additionally, at a process block 805, the process 800generates, with the processor 201 or the processor 251, user-specificbaseline data based upon the baseline sensor data and the baseline imagecapture data. Moreover, at a process block 806, the process 800generates, with the processor 201 or the processor 251, one or morepredetermined biomechanical rules according to the user-specificbaseline data. At a process block 807, the process 800 generates, withthe processor 201 or the processor 251, a virtual avatar 501 based uponthe one or more predetermined biomechanical rules. Also, at a processblock 808, the process 800 generate, with the processor 201 or theprocessor 251, one or more cues 510 corresponding to the virtual avatar501 based upon non-compliance of one or more real-world movements of thefirst user 301 with the one or more predetermined rules. Finally, at aprocess block 809, the process 800 stores, at the memory device 202 orthe memory device 252, the user-specific baseline data in a baselinedata structure 107 for access by the processor 201 or the processor 251.

A computer is intended herein to include any device that has aspecialized processor as described above. For example, a computer may bea personal computer (“PC”), laptop computer, set top box, cell phone,smartphone, tablet device, smart wearable device, portable media player,video player, etc. The configurations provided for herein may beimplemented in various forms of computers.

It is understood that the apparatuses, systems, computer programproducts, and processes described herein may also be applied in othertypes of apparatuses, systems, computer program products, and processes.Those skilled in the art will appreciate that the various adaptationsand modifications of the embodiments of the apparatuses described hereinmay be configured without departing from the scope and spirit of thepresent apparatuses, systems, computer program products, and processes.Therefore, it is to be understood that, within the scope of the appendedclaims, the present apparatuses, systems, computer program products, andprocesses may be practiced other than as specifically described herein.

I claim:
 1. A system comprising: a sensor configured to sense baselinesensor data corresponding to one or more baseline movements of a userand senses dynamic sensor data corresponding to one or more dynamicmovements of the user, the dynamic sensor data being captured after thebaseline sensor data; an image capture device integrated within a mobilecomputing device, the image capture device configured to capture onebaseline image capture data corresponding to the one or more baselinemovements of the user and configured to capture dynamic image capturedata corresponding to the one or more dynamic movements of the user, thedynamic image capture data being captured after the baseline imagecapture data, the mobile computing device being in operablecommunication with the sensor; a processor that is programmed togenerate user-specific baseline data based upon the baseline sensor dataand the baseline image capture data, generate one or more predeterminedbiomechanical rules according to the user-specific baseline data,generate a virtual avatar based upon the one or more predeterminedbiomechanical rules, generate one or more cues corresponding to thevirtual avatar based upon non-compliance of one or more real-worldmovements of the user with the one or more predetermined rules, filterthe baseline sensor data according to an optimal sensor-specific planethat intersects the user during the one or more baseline movements,filter the baseline image capture data according to an optimal imagecapture device-specific plane that intersects the user during the one ormore baseline movements, and composite the remaining data as thebaseline data for generation of the virtual avatar, the one or morereal-world movements being determined from the dynamic sensor data andthe dynamic image capture data, the virtual avatar having virtualmovements that are synchronized in real-time with the corresponding oneor more real-world movements; and a memory device configured to storethe user-specific baseline data in a baseline data structure for accessby the processor.
 2. The system of claim 1, wherein the processor isfurther programmed to render the virtual avatar on a display devicecorresponding to the mobile computing device.
 3. The system of claim 2,wherein the processor is further programmed to render a graphical userinterface on the display device that provides a menu, the menu providean indicium that activates recording of the one or more baselinemovements and an indicium that activates recording of the one or moredynamic movements.
 4. The system of claim 3, wherein the menu furthercomprises a biomechanical analysis indicium that generates access to areport of compliance of the user with the one or more predeterminedbiomechanical rules.
 5. The system of claim 4, wherein the reportincludes one or more scores corresponding to performance metrics.
 6. Thesystem of claim 3, wherein the menu further comprises a biomechanicalrecommendation indicium that generates one or more recommendations forcompliance with the one or more predetermined rules.
 7. The system ofclaim 1, wherein the one or more cues are visual cues that visuallyindicate a biomechanical structure of the user that is non-compliantwith the one or more predetermined rules.
 8. The system of claim 1,wherein the one or more cues are auditory cues that indicate, via anauditory output, a biomechanical structure of the user that isnon-compliant with the one or more predetermined rules.
 9. The system ofclaim 1, wherein the mobile computing device is selected from the groupconsisting of: a smartphone, tablet device, smart glasses, and a smartwatch.
 10. The system of claim 1, wherein the one or more baselinemovements correspond to one or more exercises that the processoranalyzes to determine one or more ranges of motion particular to theuser, wherein the one or more dynamic movements correspond to one ormore sport-specific movements, the one or more exercises being distinctfrom the one or more sport-specific movements.
 11. A computer programproduct comprising a non-transitory computer readable medium having acomputer readable program stored thereon, wherein the computer readableprogram when executed on a computer causes the computer to: sense, witha sensor, baseline sensor data corresponding to one or more baselinemovements of a user; sense, with the sensor, dynamic sensor datacorresponding to one or more dynamic movements of the user, the dynamicsensor data being captured after the baseline sensor data; capture, withan image capture device integrated within a mobile computing device,baseline image capture data corresponding to the one or more baselinemovements of the user; capture, with the image capture device, dynamicimage capture data corresponding to the one or more dynamic movements ofthe user, the dynamic image capture data being captured after thebaseline image capture data, the mobile computing device being inoperable communication with the sensor; generate, with a processor,user-specific baseline data based upon the baseline sensor data and thebaseline image capture data; generate, with the processor, one or morepredetermined biomechanical rules according to the user-specificbaseline data; generate, with the processor, a virtual avatar based uponthe one or more predetermined biomechanical rules; generate, with theprocessor, one or more cues corresponding to the virtual avatar basedupon non-compliance of one or more real-world movements of the user withthe one or more predetermined rules, the one or more real-worldmovements being determined from the dynamic sensor data and the dynamicimage capture data, the virtual avatar having virtual movements that aresynchronized in real-time with the corresponding one or more real-worldmovements; filter, with the processor, the baseline sensor dataaccording to an optimal sensor-specific plane that intersects the userduring the one or more baseline movements; filter, with the processor,the baseline image capture data according to an optimal image capturedevice-specific plane that intersects the user during the one or morebaseline movements; composite, with the processor, the remaining data asthe baseline data for generation of the virtual avatar; and store, at amemory device, the user-specific baseline data in a baseline datastructure for access by the processor.
 12. The computer program productof claim 11, wherein the computer is further caused to render thevirtual avatar on a display device corresponding to the mobile computingdevice.
 13. The computer program product of claim 12, wherein thecomputer is further caused to render, with the processor, a graphicaluser interface on the display device that provides a menu, the menuprovide an indicium that activates recording of the one or more baselinemovements and an indicium that activates recording of the one or moredynamic movements.
 14. The computer program product of claim 13, whereinthe menu further comprises a biomechanical analysis indicium thatgenerates access to a report of compliance of the user with the one ormore predetermined biomechanical rules.
 15. The computer program productof claim 14, wherein the report includes one or more scorescorresponding to performance metrics.
 16. The computer program productof claim 13, wherein the menu further comprises a biomechanicalrecommendation indicium that generates one or more recommendations forcompliance with the one or more predetermined rules.
 17. The computerprogram product of claim 11, wherein the one or more cues are visualcues that visually indicate a biomechanical structure of the user thatis non-compliant with the one or more predetermined rules.
 18. Thecomputer program product of claim 11, wherein the one or more cues areauditory cues that indicate, via an auditory output, a biomechanicalstructure of the user that is non-compliant with the one or morepredetermined rules.